System and method for displaying anonymously annotated physical exercise data

ABSTRACT

The present invention relates to a method for displaying anonymously annotated physical exercise data to a person undertaking exercises. Based on physical exercise data, the physical exercise data is annotated at a physically separate annotation unit. At the location of the person, visual recordings of the person undertaking exercises together with synchronized annotation information are displayed to the person. A system for performing the method comprises a physical data processing unit ( 1 ), a display device ( 2 ), at least one posture recording device ( 3, 3 ′), a visual recording device ( 4 ), a data storage unit ( 5 ) and a physically separate annotation unit ( 6 ) in connection with the physical data processing unit ( 1 ), the connection being via an interconnected computer network ( 7 ).

BACKGROUND OF THE INVENTION

The present invention relates to a system and method for displayinganonymously annotated physical exercise data to a person undertakingexercises.

Home rehabilitation exercises for persons suffering from a medicalcondition like a stroke or home training exercises for persons wishingto improve body motions like a golf swing can be recorded via sensors.The exercises can also be evaluated by a professional such as aphysiotherapist or a golf instructor in order to give the person adirect feedback.

If the professional performing the review is not present during theexercise, video camera recordings could be sent to him. These recordingscould be reviewed intuitively by the professional and the commentedrecordings could be understood intuitively by the person undertaking theexercise. However, these recordings, especially when sent away to aremote professional, could breach the privacy of the person.Furthermore, a completely automatic processing of such recorded imagesto provide meaningful feedback is a demanding task.

Alternatively, the sole transmission of data from the sensors would notviolate the privacy of the person. In this respect, U.S. Pat. No.6,817,979 B2 relates to a system and method which provide forinteracting with a virtual physiological model of a user with the use ofa mobile communication device. Physiological data associated with theuser is acquired from the user. The physiological data is transmitted tothe mobile communication device, preferably with the use of a wirelesscommunication protocol. The methodology further involves using themobile communication device to communicate the physiological data to anetwork server. The physiological data is integrated into the virtualphysiological model of the user. The user can access data and depictionsof the user developed from the physiological data.

By way of example, a user can create an avatar representative of thecurrent physical state of the user. The user can adjust the avatar tochange the appearance of the avatar to a more desired appearance. Forexample, the anatomical dimensions of the avatar can be changed toreflect desired waist, chest, upper arms and thigh dimensions. Givendifferences between the desired avatar features and present avatarfeatures, various training, diet and related fitness recommendations canbe developed to establish a training regimen most suited to help theuser achieve the desired fitness goals. Physiological data issubsequently acquired and applied to the user's avatar, and compared tothe desired avatar's data to determine if the training regimen iseffective in achieving the desired fitness goals.

However, in general the interpretation of sensor signals in the frontendleads to difficulties on the part of the user. It is hard to relate toan abstract rendering of an artificial screen character.

Despite this effort accordingly there still exists a need in the art fora system and a method for displaying anonymously annotated physicalexercise data to a person undertaking exercises.

SUMMARY OF THE INVENTION

To achieve this and other objects the present invention is directed to amethod for displaying anonymously annotated physical exercise data to aperson undertaking exercises, comprising the steps of:

a) gathering physical exercise data from a person undertaking exercises;b) synchronously gathering visual recordings of the person undertakingexercises;c) transmitting the physical exercise data to a physically separateannotation unit;d) based on the physical exercise data, annotating the physical exercisedata at the physically separate annotation unit;e) transmitting the annotation information to a display and processingunit for review of the person undertaking exercises;f) displaying the visual recordings of the person undertaking exercisestogether with synchronized annotation information to the person.

DETAILED DESCRIPTION OF THE INVENTION

Before the invention is described in detail, it is to be understood thatthis invention is not limited to the particular component parts of thedevices described or process steps of the methods described as suchdevices and methods may vary. It is also to be understood that theterminology used herein is for purposes of describing particularembodiments only, and is not intended to be limiting. It must be notedthat, as used in the specification and the appended claims, the singularforms “a,” “an” and “the” include singular and/or plural referentsunless the context clearly dictates otherwise.

In the context of the present invention, the term anonymously annotateddata denotes data where a third person performing the annotation has noknowledge about the identity of the person whose data he is annotating.In particular, the data does not allow for a recognition of the person.One way of achieving the anonymization is by assigning identificationnumbers to the data. Physical exercise data is data relating tomovements or other exercises of a person.

The first two steps of the method describe how two different sets ofinformation about the exercise of the person are gathered. Firstly,physical exercise data is gathered, for example by continuouslymonitoring sensor signals from the person. At the same time, visualrecordings are gathered, for example by using a digital video camera. Bysynchronously gathering this data it is ensured that later on, a certainportion of the video stream can be attributed to a certain portion ofthe sensor signal stream and vice versa.

As the visual recordings and the physical exercise data are separateentities, the physical exercise data can then be transmitted to aphysically separate annotation unit. The physical separation of theannotation unit provides for an anonymization of the data. At theannotation unit the physical exercise data can be processed intorepresentations of the exercise for review by a third person. Thephysical exercise data can then be annotated. This includes automaticprocessing of the data, for example by detecting deviations from motiontemplates. Furthermore, the third person can include comments andsuggestions to provide helpful feedback to the person performing theexercise. Afterwards, the annotation information is transmitted to adisplay and processing unit at the site of the person performing theexercise. Here, the annotation information is joined with the visualrecordings. The recordings of the person undertaking exercises are thendisplayed to the person together with the synchronized annotationinformation. The synchronization provides for displaying the annotationat the correct time so the person can directly understand what hascaught the attention of the reviewer or the automatic reviewing system.

In summary, with the method according to the present invention anexercise of a person can be reviewed anonymously and feedback can begiven to the person. The anonymization allows for the sharing ofprofessional resources, making the reviewing process more efficient. Atthe same time, when the person receives the feedback it is very clearlyshown to him, via the visual recordings, which part of the exercise hasprompted the feedback.

In one embodiment of the invention, at the physically separateannotation unit in step d) an avatar is calculated based on the physicalexercise data. For the purposes of this invention, the term ‘avatar’shall denote a computer-generated abstract rendering which representsthe posture or motions of a person. In simple cases, the avatar may be astick figure. In more sophisticated cases, the avatar may representadditional information like the pulse rate, the amount of sweating,muscle fatigue and the like. An advantage of using an avatarrepresentation is that the avatar can be rotated on the screen of theannotation unit while representing the exercise. This enables thereviewer to choose the best viewing angle for assessing the exercise.

In a further embodiment of the invention step f) additionally comprisescalculating an avatar and displaying the avatar synchronously with thevisual recordings and the annotations to the person. In summary, theperson will then see the visual recording of his exercise, theannotations and the avatar. This is advantageous as the avatar maydepict more clearly the motions of the persons if they are obscured inthe visual recording by baggy clothing or if they have not been recordedcorrectly on camera. Again, the avatar may be rotated to achieve thebest viewing perspective. Another option is to provide multiple viewingangles with one or more avatars.

In a further embodiment of the invention transmitting the physicalexercise data in step c) and transmitting the annotation information instep e) is undertaken via an interconnected computer network, preferablythe internet. This allows a remotely located person to perform thereview and the annotation. Suitable protocols can include those of theTCP/IP protocol.

In a further embodiment of the invention the physical exercise data fromthe person is selected from the group comprising motion data, posturedata, electromyographic data, pulse rate, blood pressure, blood oxygencontent, blood sugar content, severity of perspiration and/orrespiratory rate. These data types either relate to the exercise itself,such as in the case of motion and posture data. Other data types relateto the overall condition or physical fitness of the person. Knowledgeabout this can give valuable insight into the effectiveness ofrehabilitation or training measures. For example, it may be inferredwhether the person is in the supercompensation phase after a trainingstimulus.

In a further embodiment of the invention the annotation information isselected from the group comprising visual information, audio signalsand/or speech recordings. Visual information can be in the form ofmarkings such as arrows pointing out a specific issue that are insertedinto the images of the avatar. Additionally, small video clips can beinserted to show the correct execution of the exercise. Other visualinformation can be written comments or graphs showing statistics of datalike electromyographic data, pulse rate, blood pressure, blood oxygencontent, blood sugar content, severity of perspiration and/or therespiratory rate. This enables to assess the situation of the personperforming the exercise at one glance. Audio signals can be simple beepswhen a movement is no performed correctly. Recorded speech comments canbe added by the reviewer when this is the simplest way of explaining anexercise.

The present invention is further directed towards a system fordisplaying anonymously annotated physical exercise data to a personundertaking exercises, comprising:

-   -   a physical data processing unit;    -   a display device in communication with the physical data        processing unit;    -   at least one posture recording device assigned to the person        undertaking exercises and in communication with the physical        data processing unit;    -   a visual recording device in communication with the physical        data processing unit;    -   a data storage unit for storing and retrieving data from the        physical data processing unit and the visual recording device;        the data storage means being in communication with the physical        data processing unit;    -   a physically separate annotation unit in connection with the        physical data processing unit, the connection being via an        interconnected computer network.

In one embodiment of the invention the at least one posture recordingdevice comprises a motion sensor on the person undertaking exercises,the sensor being selected from the group comprising accelerationsensors, inertia sensors and/or gravity sensors. The motion sensors canbe worn on the body of the person on selected locations like upper arm,lower arm, upper leg, lower leg or torso. They can be commerciallyavailable highly integrated solid state sensors. The transmission of thesensor signals to the posture assessment unit can be undertaken viawire, wirelessly or in a body area network using the electricalconductivity of the human skin. After calculation of the person'sposture the result can be present in the form of an avatar.

In a further embodiment of the invention the at least one posturerecording device comprises an optical mark on the person undertakingexercises. The posture recording device then employs an optical trackingsystem for tracking the at least one optical mark. Based on the signalsof the optical tracking system a representation of the person's postureis then calculated. The optical marks can be borne on the body of theperson on selected locations like upper arm, lower arm, upper leg, lowerleg or torso. The tracking of the marks can be effected with a singlecamera or a multitude of cameras. When a stereo camera is used,three-dimensional posture and movement data is generated. After imageprocessing and calculation of the person's posture the result can bepresent in the form of an avatar.

It is also possible to combine several posture monitoring principles.For example, a combination of motion sensors and optical tracking mayprovide complementary data to better calculate the posture of theperson.

A further aspect of the present invention is the use of a systemaccording to the present invention claims for displaying anonymouslyannotated physical exercise data to a person undertaking exercises.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will become more readily understood with referenceto the following drawing, wherein

FIG. 1 shows a system according to the present invention

FIG. 2 shows a synchronous overlay of visual recordings and an avatarrepresenting physical exercise data

FIG. 3 shows a flowchart of a method according to the present invention

FIG. 4 shows modules for performing a method according to the presentinvention.

DETAILED DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a system according to the present invention for displayinganonymously annotated physical exercise data to a person undertakingexercises. As posture recording devices, the person has motion sensors 3situated on his thighs and his ankles. Furthermore, optical marks 3′ arelocated on the wrist and the torso. Being physical exercise data, thesignals of the motion sensors 3 are transmitted wirelessly to thephysical data processing unit 1 where the raw sensor signals areprocessed into motion and posture data. A video camera 4 records themotions of the person. Furthermore, the physical data processing unit 1performs optical tracking operations on the video stream of the camera 4for identifying the position and the movement of the optical marks 3′.This is also processed into motion and posture data and complements thedata obtained from the motion sensors 3.

The raw or processed sensor signals and positional information from theoptical marks 3′ are stored in a data storage unit 5. Furthermore, thevideo stream of the person performing the exercise is also stored there.The data in the data storage unit 5 is stored together with aninformation about the time of recording. This makes it possible tocorrelate or synchronize the information, for example knowing whichposition as indicated by posture recording devices 3, 3′ corresponds towhich frame of a video clip of the person performing the exercise.

Using an interconnected computer network such as the internet 7, thephysical data processing unit 1 transmits the processed sensor 3 signalsand positional information from the optical marks 3′ to a physicallyseparate annotation unit 6. Temporal information is also transmitted.This annotation unit then calculates a visual representation such as anavatar from the received physical data. A physical therapist views themotion of the visual representation on his terminal 8 and commentssequences, thus performing the annotation. The annotation together withthe time within the exercise when the annotation has been made istransmitted back to the physical data processing unit 1 at the locationof the person undertaking exercises. Again, the transmission is achievedover an interconnected computer network such as the internet 7.

The physical data processing unit 1 then accesses the data storage unit5 and retrieves the recorded data and video clips from the particularexercise that has been annotated. A movie sequence is generated forviewing by the person and displayed on display 2. In this case, thevideo stream of the person and an avatar calculated from the recordeddata are shown simultaneously. At the appropriate time, the comments ofthe physical therapist are also displayed or voiced to the person.

FIG. 2 shows a synchronous overlay of visual recordings and an avatarrepresenting physical exercise data. A person has been performing anexercise. Physical data representing his motions has been recorded andused for calculation of an avatar representation. The avatar's motionhas been time-resolved and split into a stream of individual frames 20.Likewise, the person's movements have been recorded by a video camera.This video image sequence has also been time-resolved and split into astream of individual frames 21. As the physical exercise data and thevisual recordings have been gathered simultaneously, one common timeline can be assigned to them. The time line in FIG. 2 beneath the framestreams arbitrarily begins at 4:16 minutes and ends at 4:21 minutes.

In the exercise of FIG. 2, the person starts with both of his armsstretched lowered. In the images, the left arm is kept stretched andraised along the coronal plane until the hand is above the person'shead. The arm is kept in this position while the same movement issupposed to be performed with the right arm. At the time of 4:20, theperson is not able to keep his right arm outstretched in the horizontalposition. The arm is bent at the elbow. This makes it much easier tolift the arm so at this point no therapeutic benefit is gained. Aphysical therapist remotely reviewing the avatar frames 20 can thensingle out the frame at 4:20 minutes and add a visual or verbal comment.This comment, together with the information that it is to be shown at4:20 minutes into the exercise, is transmitted to the person for futurereviewal. At the person's location the annotation can then be combinedwith the visual recordings 21 so that the person can relate moredirectly to the exercise and contemplate his errors in performing it.

FIG. 3 shows a flowchart of a method according to the present invention.The first step 30 is to record the exercise a person is performingvisually, using a camera, and via posture data, using sensors. Thevisual recordings are stored 31 and the posture recordings aretransmitted to an annotation system 32. Using the annotation system, aperson reviews the posture recordings and adds his comments and markers33. These annotations are transmitted back to the patient system 34,wherein ‘patient’ denotes the person performing an exercise. On thepatient side, the stored visual recordings are retrieved 35 and combinedwith the annotations 36 in order to give the person a comprehensivefeedback that still does not compromise his anonymity.

FIG. 4 shows modules for performing a method according to the presentinvention to complement the depiction of a system in FIG. 1. A sensorreceiver 40 receives signals from motion sensors or information from thetracking of optical marks. This sensor receiver 40 communicates its datato a movement transmission module 41. Synchronously with the sensorreceiver 40, a camera 42 captures a video sequence of the personperforming exercises. These video sequences are stored in a storagefacility 43. The movement transmission module 41 transmits its data to aremotely located movement receiver 45. This is symbolized by barrier 44separating the two sub-groups of modules.

The movement receiving module 45 passes the data on to a movementannotator 46 where the data is transformed into processible data andannotated by a reviewer. The annotation together with information on thetemporal position of the annotation within the exercise is passed on toannotation transmission module 47. Aforementioned annotationtransmission module 47 transmits the information to an annotationreceiver 48 located at the sub-group of modules assigned to the personperforming the exercise. This annotation information reaches aprocessing and overlay module 49 which accesses video sequences from thestorage module 43 and combines the sequences with the annotation so thatthe annotation is present at the appropriate time of the video sequence.Finally, via a rendering module 50, the overlaid video sequence isdisplayed to the person who has performed the exercise.

To provide a comprehensive disclosure without unduly lengthening thespecification, the applicant hereby incorporates by reference each ofthe patents and patent applications referenced above.

The particular combinations of elements and features in the abovedetailed embodiments are exemplary only; the interchanging andsubstitution of these teachings with other teachings in this and thepatents/applications incorporated by reference are also expresslycontemplated. As those skilled in the art will recognize, variations,modifications, and other implementations of what is described herein canoccur to those of ordinary skill in the art without departing from thespirit and the scope of the invention as claimed. Accordingly, theforegoing description is by way of example only and is not intended aslimiting. The invention's scope is defined in the following claims andthe equivalents thereto. Furthermore, reference signs used in thedescription and claims do not limit the scope of the invention asclaimed.

1. A method for displaying anonymously annotated physical exercise datato a person undertaking exercises, comprising the steps of: a) gatheringphysical exercise data from a person undertaking exercises; b)synchronously gathering visual recordings of the person undertakingexercises; c) transmitting the physical exercise data to a physicallyseparate annotation unit; d) based on the physical exercise data,annotating the physical exercise data at the physically separateannotation unit; e) transmitting the annotation information to a displayand processing unit for review of the person undertaking exercises; f)displaying the visual recordings of the person undertaking exercisestogether with synchronized annotation information to the person. 2.Method according to claim 1, wherein at the physically separateannotation unit in step d) an avatar is calculated based on the physicalexercise data.
 3. Method according to claim 1, wherein step f)additionally comprises calculating an avatar and displaying the avatarsynchronously with the visual recordings and the annotations to theperson.
 4. Method according to claim 1, wherein transmitting thephysical exercise data in step c) and transmitting the annotationinformation in step e) is undertaken via an interconnected computernetwork, preferably the internet.
 5. Method according to claim 1,wherein the physical exercise data from the person is selected from thegroup comprising motion data, posture data, electromyographic data,pulse rate, blood pressure, blood oxygen content, blood sugar content,severity of perspiration and/or respiratory rate.
 6. Method according toclaim 1, wherein the annotation information is selected from the groupcomprising visual information, audio signals and/or speech recordings.7. A system for displaying anonymously annotated physical exercise datato a person undertaking exercises, comprising: a physical dataprocessing unit (1); a display device (2) in communication with thephysical data processing unit (1); at least one posture recording device(3, 3′) assigned to the person undertaking exercises and incommunication with the physical data processing unit (1); a visualrecording device (4) in communication with the physical data processingunit (1); a data storage unit (5) for storing and retrieving data fromthe physical data processing unit (1) and the visual recording device(4); the data storage means (5) being in communication with the physicaldata processing unit (1); a physically separate annotation unit (6) inconnection with the physical data processing unit (1), the connectionbeing via an interconnected computer network (7).
 8. System according toclaim 7, wherein the at least one posture recording device (3, 3′)comprises a motion sensor (3) on the person undertaking exercises, thesensor being selected from the group comprising acceleration sensors,inertia sensors and/or gravity sensors.
 9. System according to claim 7,wherein the at least one posture recording device (3, 3′) comprises anoptical mark (3′) on the person undertaking exercises.
 10. Use of asystem according to claim 7 for displaying anonymously annotatedphysical exercise data to a person undertaking exercises.